Beating Atari with Natural Language Guided Reinforcement Learning

We introduce the first deep reinforcement learning agent that learns to beat Atari games with the aid of natural language instructions. The agent uses a multimodal embedding between environment observations and natural language to self-monitor progress through a list of English instructions, granting itself additional reward for completing instructions in addition to increasing the game score. Our agent significantly outperforms Deep-Q Networks, Asynchronous Advantage Actor-Critic (A3C) agents, and the best agents posted to OpenAI Gym [4] on what is often considered the hardest Atari 2600 environment [2]: MONTEZUMA’S REVENGE. Videos of Trained MONTEZUMA’S REVENGE Agents: Our Best Current Model. Score 3500. Best Model Currently on OpenAI Gym. Score 2500. Standard A3C Agent Fails to Learn. Score 0. Figure 1: Left: an agent exploring the first room of MONTEZUMA’S REVENGE. Right: an example list of natural language instructions one might give the agent. The agent grants itself an additional reward after completing the current instruction. “Completion” is learned by training a generalized multimodal embedding between game images and text.